The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations

Background: Cronbach's alpha is widely used as the preferred index of reliability for medical postgraduate examinations. A value of 0.8-0.9 is seen by providers and regulators alike as an adequate demonstration of acceptable reliability for any assessment. Of the other statistical parameters, Standard Error of Measurement (SEM) is mainly seen as useful only in determining the accuracy of a pass mark. However the alpha coefficient depends both on SEM and on the ability range (standard deviation, SD) of candidates taking an exam. This study investigated the extent to which the necessarily narrower ability range in candidates taking the second of the three part MRCP(UK) diploma examinations, biases assessment of reliability and SEM.Methods: a) The interrelationships of standard deviation (SD), SEM and reliability were investigated in a Monte Carlo simulation of 10,000 candidates taking a postgraduate examination. b) Reliability and SEM were studied in the MRCP(UK) Part 1 and Part 2 Written Examinations from 2002 to 2008. c) Reliability and SEM were studied in eight Specialty Certificate Examinations introduced in 2008-9.Results: The Monte Carlo simulation showed, as expected, that restricting the range of an assessment only to those who had already passed it, dramatically reduced the reliability but did not affect the SEM of a simulated assessment. The analysis of the MRCP(UK) Part 1 and Part 2 written examinations showed that the MRCP(UK) Part 2 written examination had a lower reliability than the Part 1 examination, but, despite that lower reliability, the Part 2 examination also had a smaller SEM (indicating a more accurate assessment). The Specialty Certificate Examinations had small Ns, and as a result, wide variability in their reliabilities, but SEMs were comparable with MRCP(UK) Part 2.Conclusions: An emphasis upon assessing the quality of assessments primarily in terms of reliability alone can produce a paradoxical and distorted picture, particularly in the situation where a narrower range of candidate ability is an inevitable consequence of being able to take a second part examination only after passing the first part examination. Reliability also shows problems when numbers of candidates in examinations are low and sampling error affects the range of candidate ability. SEM is not subject to such problems; it is therefore a better measure of the quality of an assessment and is recommended for routine use

Oncoplastic breast consortium recommendations for mastectomy and whole breast reconstruction in the setting of post-mastectomy radiation therapy

Aim: Demand for nipple-and skin-sparing mastectomy (NSM/SSM) with immediate breast reconstruction (BR) has increased at the same time as indications for post-mastectomy radiation therapy (PMRT) have broadened. The aim of the Oncoplastic Breast Consortium initiative was to address relevant questions arising with this clinically challenging scenario. Methods: A large global panel of oncologic, oncoplastic and reconstructive breast surgeons, patient advocates and radiation oncologists developed recommendations for clinical practice in an iterative process based on the principles of Delphi methodology. Results: The panel agreed that surgical technique for NSM/SSM should not be formally modified when PMRT is planned with preference for autologous over implant-based BR due to lower risk of long-term complications and support for immediate and delayed-immediate reconstructive approaches. Nevertheless, it was strongly believed that PMRT is not an absolute contraindication for implant-based or other types of BR, but no specific recom-mendations regarding implant positioning, use of mesh or timing were made due to absence of high-quality evidence. The panel endorsed use of patient-reported outcomes in clinical practice. It was acknowledged that the shape and size of reconstructed breasts can hinder radiotherapy planning and attention to details of PMRT techniques is important in determining aesthetic outcomes after immediate BR. Conclusions: The panel endorsed the need for prospective, ideally randomised phase III studies and for surgical and radiation oncology teams to work together for determination of optimal sequencing and techniques for PMRT for each patient in the context of BRPeer reviewe

Writing clinical scenarios for clinical science questions

Written knowledge assessments for physicians in training typically involve multiple-choice questions that use a clinical scenario in a single-best-answer format. The Royal College of Physicians Part 1 MRCP(UK) examination includes basic sciences themes that are challenging to assess through a clinical scenario. A realistic clinical setting based on everyday clinical practice and integral to the question is the clearest demonstration that the knowledge being assessed is clinically relevant. However, without special attention to detail, the scenario in a clinical science question can appear redundant or artificial. Reading unnecessary material frustrates candidates and threatens the reputation of the assessment. In this paper we discuss why a clinical scenario is important for basic science questions and offer advice on setting realistic and plausible clinical scenarios for such questions